A Numerical Framework for Solving Discrete and Finite Markov Models

Discrete time Markov chains (DTMCs) are commonly described by a state transition matrix that contains sufficient information to figure out the stationary state distribution of the Markov model. In this work, we propose recursive stochastic equations as an alternative description that clearly separates the functional behavior of the model from its random influences that determine the state transition matrix. For the computation of the stationary state distribution, several optimization methods are proposed that take partly advantage of the new specification scheme and that can be applied both to aperiodic and periodic Markov chains. We have written a compiler that converts the recursive equations with all optimization steps in a numerical program to evaluate even large and multi-dimensional Markov models in a short time.